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Routing in a Multi-provider Internet 
Status of this Memo 


This memo provides information for the Internet community. This memo 
does not specify an Internet standard of any kind. Distribution of 
this memo is unlimited. 


Abstract 


This document was prepared by the author on behalf of the Internet 
Architecture Board (IAB). It is offered by the IAB to stimulate 
discussion. 


Over the past few years the Internet has undergone significant 
changes. Among them is the emergence of multiple Network Service 
Providers, where resources that provide Internet-wide IP connectivity 
(routers, links) are controlled by different organizations. This 
document presents some of the issues related to network layer routing 
in a multi-provider Internet, and specifically to the unicast 
routing. 


1. Network Service Providers vs Network Service Subscribers 


Within the current routing paradigm the service offered by a provider 
at the network layer (IP) is the set of destinations (hosts) that can 
be reached through the provider. Once a subscriber establishes direct 
connectivity to a provider, the subscriber can in principle reach all 
the destinations reachable through the provider. Since the value of 
the Internet-wide connectivity service offered by a provider 
increases with the number of destinations reachable through the 
provider, providers are motivated to interconnect with each other. 


In principle a provider need not offer the same service (in terms of 
the set of destinations) to all of its subscribers -- for some of the 
subscribers the provider may restrict the services to a subset of the 
destinations reachable through the provider. In fact, for certain 
types of subscribers constrained connectivity could be seen as part 
of the service offered by a provider. 


In a multi-provider environment individual providers may be driven by 


diverse and sometimes even conflicting goals and objectives. Some of 
the providers exist to provide connectivity to only a specific group 
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of Network Service Subscribers. Other providers place no constraints 
on the subscribers that can subscribe to them, as long as the 
subscribers pay the fee charged by the providers. Some of the 
providers place certain constraints on the reselling of the 
connectivity services by organizations (e.g., other providers) 
attached to the providers. Some of the providers may be operated by 
companies that are subject to specific regulations (e.g., regulated 
monopoly), while other providers are completely unregulated. The 
scope of geographical coverage among providers varies from a small 
region (e.g., county, town) to a country-wide, international, or even 
intercontinental. 


There is no centralized control over all the providers in the 
Internet. The providers do not always coordinate their efforts with 
each other, and quite often are in competition with each other. 


Despite all the diversity among the providers, the Internet-wide IP 
connectivity is realized via Internet-wide distributed routing, which 
involves multiple providers, and thus implies certain degree of 
cooperation and coordination. Therefore, there is a need to balance 
the providers’ goals and objectives against the public interest of 
Internet-wide connectivity and subscribers’ choices. Further work is 
needed to understand how to reach the balance. 


2. Routing Requirements 


Conceptually routing requirements can be classified into the 
following three categories: source preferences, destination 
preferences, and constraints on transit traffic. Source preferences 
allow an originator of a packet to exert control over the path to a 
destination. Destination preferences allow a destination to exert 
control over the path from a source to the destination. Constraints 
on transit traffic allow a provider to control the traffic that can 
traverse through the resources (routers, links) controlled by the 
provider. 


From a conceptual point of view the requirements over the degree of 
control for source and destination preferences may vary from being 
able to just provide connectivity (regardless of the path), to being 
able to select immediate providers, to more complex scenarios, where 
at the other extreme a subscriber may want to have complete control 
over the path selection. 


From a conceptual point of view the requirements over the degree of 
control for transit traffic may vary from control based only on the 
direct physical connectivity (controlling the set of organizations 
directly connected to the provider), to being able to restrict 
traffic to a particular set of sources or destinations, or a 
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combination of particular sources and destinations, or even take into 
account the paths to/from these sources and/or destinations. 


In view of a potentially wide variety of routing requirements, we 
need to get a better understanding on the relative practical 
importance of various routing requirements. In practice organizations 
usually don’t formulate their routing requirements in a vacuum. For 
example, since the primary role of a provider is to provide services 
to a set of subscribers, the provider usually formulates its routing 
requirements based on the set of the routing requirements of the 
subscribers the provider is expected to serve. 


Support for various routing requirements should take into account the 
overhead and the scope of the overhead associated with those 
requirements. A situation where an organization can unilaterally 
impose routing information overhead on other organization (e.g., by 
requiring the other organization to maintain an additional routing 
information) should be viewed as undesirable. The cost of supporting 
a particular routing requirement should not be borne by organizations 
that do not benefit from supporting that requirement. Ideally the 
routing system should allow to shift the overhead associated with a 
particular routing requirement towards the entity that instigates the 
requirement (for example, there is a need to carefully balance the 
overhead associated with maintaining a state needed for multi-hop 
header compression vs carrying explicit forwarding information on a 
per packet basis). Organizations with simple routing requirements 
shouldn’t bear the same routing information overhead as organizations 
with complex routing requirements. 


A situation where the overhead associated with supporting a 
particular routing requirement has to be carried by every entity 
(e.g., router, host) within an organization that would like to impose 
the requirement could be viewed as undesirable. An organization 
should be able to instantiate its routing requirements in a more or 
less central fashion, for example by utilizing just some of the 
routers. 


Even if the scope of the routing information overhead is purely 
local, there is a need to perform a careful analysis of the tradeoff 
between the potential benefits and the cost associated with 
supporting various routing requirements. 


3. Encapsulation 
The technique of encapsulation allows for the creation of a "virtual" 


IP overlay over an existing IP infrastructure. This has certain 
implications for the Internet routing system. 
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In the presence of encapsulation, a provider may no longer be able to 
constrain its transit traffic to a particular set of ultimate sources 
and/or destinations, as a packet may be encapsulated by some router 
along the path, with the original source and/or destination addresses 
being "hidden" (via encapsulation) at the Network layer. Likewise, 
encapsulation may affect source and destination preferences, as a 
source (or a destination) may either (a) be unaware of the 
encapsulation, or (b) have little or no control over the encapsulated 
segment of a path. 


Further work is needed to understand the implications of the overlay 
capabilities created via encapsulation on the semantics of routing 
requirements, as well as the interaction among the routing 
requirements by the entities that form the overlay and the entities 
that form the underlying infrastructure. 


4. Price Structure and its Impact on Routing 


Routing among providers, as well as between providers and subscribers 
may be influenced by the price structure employed by the providers, 
as well as the usage pattern of the subscribers. A provider can view 
routing as a mechanism that allows the provider to exert control over 
who can use the provider’s services. A subscriber can view routing as 
a mechanism that allows the subscriber to exert control over the 
price it pays for the Internet connectivity. 


The need to exert control has to be carefully balanced against the 
cost of the routing mechanisms needed to provide such control. Ina 
competitive market one could question the viability of a mechanism 
whose incremental cost would be greater than the saving recovered by 
the mechanism -- competitive pressure or alternate mechanisms are 
likely to push providers and subscribers towards choosing the 
cheapest mechanism. 


5. Scalability 


One of the key requirements imposed on the Internet routing is its 
ability to scale. In addition to conventional metrics for scalability 
(e.g., memory, CPU, bandwidth), we need to take into account 
scalability with respect to the human resources required to operate 
the system. The need for deployment of CIDR already showed that a 
routing scheme that scales linearly with respect to the number of 
connected networks, or even to the number of connected organizations 
is unacceptable today, and is likely to be unacceptable in the long 
term. It is not clear whether routing that scales linearly with the 
number of providers is going to be acceptable in the long term. 
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Scaling implies that the Internet routing system needs to have 
powerful mechanisms to provide routing information 
aggregation/abstraction. 


In the absence of Internet-wide coordination and in the presence of 
competition among the providers, the aggregation/abstraction 
mechanisms should minimize preconditions as well as limit the amount 
of required inter-provider coordination. Ideally the routing system 
should allow a provider to control the amount of its local resources 
needed to deal with the routing overhead based on considerations that 
are purely local to the provider. 


One of the side effects of the routing information 
aggregation/abstraction is that some of the routing information is 
going to be lost. This may impact route optimality and even the 
ability to find an existing route. The need for routing information 
aggregation/abstraction also implies certain homogeneity of the 
information to be aggregated/abstracted. This needs to be counter- 
balanced against the potential diversity of routing requirements. 


As a way to deal with the routing information loss due to 
aggregation/abstraction, we need to explore mechanisms that allow 
routing that is based on the on-demand acquisition of subsets of 
unaggregated information. 


The overhead associated with supporting specific routing requirements 
has a direct impact on the overall scalability of the Internet 
routing system. We need to get a better understanding of how various 
routing requirements impact scalability. When the impact is 
Significant, and the requirements have practical importance we need 
to develop mechanisms that allow the impact to be reduced. 


6. Hierarchical Routing 


Classless Inter-Domain Routing (CIDR) (RFC1518, RFC1519) that is used 
today for scalable Internet-wide routing is based on the technique of 
hierarchical routing. Essential to this technique is the assumption 
that Network layer addresses assigned to individual entities (e.g., 
hosts, routers) reflect the position of these entities within the 
network topology -- addresses are said to be "topologically 
significant". With CIDR addresses assigned to most of the individual 
sites are expected to reflect providers the sites are connected to -- 
CIDR uses "provider-based" addresses. 


One of the fundamental consequences of using hierarchical routing is 
that in order to preserve topological significance of network 
addresses, changes in the network topology may need to be accompanied 
by the corresponding changes in the addresses. Presence of multiple 
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providers serving the same geographical area implies that a 
subscriber should be able to switch from one provider to another. 
Since such a switch implies changes in the Internet topology, it 
follows that to retain topological significance of the (provider- 
based) addresses within the subscriber, the subscriber has to change 


the addresses of all of its entities -- the process known as 
"renumbering". There are already tools to facilitate this process -- 
Dynamic Host Configuration Protocol (DHCP). However, DHCP is not yet 


widely deployed. Further work is needed to improve these tools, get 
them widely deployed, and to integrate them with Domain Name System 
(DNS). 


Multi-level hierarchical routing allows for recapturing additional 
routing information (routing entropy) due to the mismatch between 
addresses and topology at a particular level in the routing hierarchy 
at some higher level in the hierarchy (e.g., at an exchange point 
among providers). This enables the routing system to contain the 
scope of entities impacted by the mismatch. Containing the scope of 
entities could be an important factor to facilitate graceful 
renumbering. Further work is needed to develop appropriate 
deployment strategies to put these capabilities in place. 


It is important to emphasize that the requirement to maintain 
topologically significant addresses doesn’t need to be applied 
indiscriminately to all the organizations connected to the Internet 
-- hierarchical routing requires that most, but not all addresses be 
topologically significant. For a large organization it could be 
sufficient if the set of destinations within the organization can be 
represented within the Internet routing system as a small number of 
address prefixes, even if these address prefixes are independent of 
the providers that the organization uses to connect to the Internet 
("provider-independent" addresses). The volume of routing information 
that a large organization would inject into the Internet routing 
system would be comparable to the (aggregated) routing information 
associated with a large number of small organizations. 


Existence of multiple providers allows a subscriber to be 
simultaneously connected to more than one provider (multi-homed 
subscribers). CIDR offers several alternatives for handling such 
cases. We need to gain more operational experience as well as better 
understand tradeoffs associated with the proposed alternatives. 


An alternative to CIDR address assignment is to assign addresses 
based purely on the geographical location. However, address 
assignment that reflects geographical location of an entity implies 
that either (a) the Internet topology needs to be made sufficiently 
congruent to the geography, or (b) addresses aren’t going to be 
topologically significant. In the former case we need to understand 


Rekhter [Page 6] 


RFC 1787 Routing in a multi-provider Internet April 1995 


the driving forces that would make the topology congruent to the 
geography. In the latter case techniques other than hierarchical 
routing need to be developed. 


7. Routing Information Sharing 


While ensuring Internet-wide coordination may be more and more 
difficult, as the Internet continues to grow, stability and 
consistency of the Internet-wide routing could significantly benefit 
if the information about routing requirements of various 
organizations could be shared across organizational boundaries. Such 
information could be used in a wide variety of situations ranging 
from troubleshooting to detecting and eliminating conflicting routing 
requirements. The scale of the Internet implies that the information 
should be distributed. Work is currently underway to establish 
depositories of this information (Routing Registries), as well as to 
develop tools that analyze, as well as utilize this information. 


8. Summary 


In this section we enumerate some of the issues that the IAB thinks 
should be brought to the attention of the Internet community. 


The following two tasks require the most immediate attention: 


- further work is needed to develop technologies that facilitate 
renumbering 


- further work is needed to investigate feasibility of routing 
information aggregation above the direct (immediate) provider 
level 


The following tasks are viewed as medium term: 


-— further work is needed to get a better understanding on the 
relative practical importance of various routing requirements 


- further work is needed to understand of how various routing 
requirements impact scalability of the routing system 


- further work is needed to investigate alternatives to 
hierarchical routing 


Finally, the following tasks are viewed as long term: 


- further work is needed to understand and utilize the benefits of 
routing information sharing 
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10. 


- further work is needed to understand the implications of virtual 
overlays created via encapsulation 


- further work is needed to understand how different price 
structures influence routing requirements 


-— further work is needed to understand how to balance the 
providers’ goals and objectives against the public interest of 
Internet-wide connectivity and subscribers’ choices. 


Conclusions 


This document presents some of the issues related to routing ina 
multi-provider Internet. There are no doubt routing-related areas 
that are not covered in this document. For instance, such areas as 
multicast routing, or routing in the presence of mobile hosts, or 
routing in the presence of a large shared media (e.g., ATM) aren’t 
discussed here. Further work is needed to understand the implications 
of a multi-provider Internet on these areas. 


The impact of multi-provider Internet goes well beyond just routing, 
and percolates into such areas as network management, 
troubleshooting, and others. Further work is needed to assess the 
implications of multi-provider environment on these areas, as well as 
to understand the interaction among all these areas from a system- 
wide perspective. 
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